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ABSTRACT 


The  design  of  three  vision-expert  programs  is  described.  The  vision  experts  are 
based  on  appearance  models  of  objects  in  a  typical  chest  radiograph  which  were 
derived  using  information  from  human  experts.  Each  expert  is  based  on  a  different 
technical  concept.  The  rib  expert  uses  relational  constraints  on  parameter  (Hough 
transform)  space  to  determine  if  a  rib  has  been  detected;  the  vascularity  expert  uses  a 
back-projected  Hough  transform;  and  the  nodule  expert  uses  features  which  were 
derived  from  studies  of  the  nodule  recognition  process  of  radiologists.  The  efficacy  of 
the  descriptive  models  and  their  implementations  are  evaluated.  The  processes  of 
development  and  implementation  of  image  experts  are  discussed. 

INTRODUCTION 

The  Automated  Nodule  Detection  System  (ANDS)  is  an  experimental  program 
for  prescreening  of  chest  radiographs  for  nodular  abnormalities  [Lampeter,  1983]. 
The  results  of  ANDS  are  presented  to  a  radiologist  for  final  inspection/diagnosis. 
ANDS  is  based  on  the  system  described  by  Ballard  and  Sklansky  [1973].  It  differs 
essentially  in  the  methods  of:  pre-processing  (spline  filtering),  rib  detection  (a  local 
technique  is  used)  [Ballard,  1978],  and  the  way  in  which  false  positives  are  rejected 
(the  subject  of  this  paper).  The  processes  of  ANDS  are  outlined  in  Fig.  1. 

The  candidate  nodule  detector  in  ANDS  is  based  on  a  simple  circle  detector 
[Ballard  and  Sklansky,  1973].  This  detector  locates  all  approximately  circular  shapes 
which  have  a  specified  radius.  Not  all  of  these  reported  shapes,  or  candidate  nodules 
(CNs),  are  in  fact  nodules.  The  goal  of  the  experts  is  to  distinguish  the  nodules  from 
the  non-nodules  which  are  comprised  of:  vascular  structures,  points  on  or  near  the 
lung  borders,  ribs,  nipples,  and  noise.  Thus,  these  experts  reduce  the  false  positive 
rate  and  present  the  user  with  a  display  of  sites  in  the  film  that  are  most  similar  to 
nodules.  Fig.  2  illustrates  a  display  of  CNs  before  and  after  processing  by  the  image 
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Figure  1.  The  Automated  Nodule  Detection  System  (ANDS).  A  chest  radiograph  is 
presented  to  ANDS  which  analyzes  it  for  the  presence  of  pulmonary  nodule  and 
which  produces  a  display  of  candidate  nodule  sites.  ANDS  incorporates  three  wsion 
experts:  a  Rib  Expert  and  a  Vascularity  Expert  provide  information  to  a  pattern 
classifier  which  classifies  a  CN;  and  a  Nodule  Expert  uses  a  set  of  rules  to  distinguish 
nodules  from  false  positives,  causing  the  obvious  false  positives  to  be  omitted  from 
the  CN  sites  that  are  presented  to  the  radiologist. 


Figure  2.  ANDS  prescreens  a  chest  radiograph,  using  image  experts  to 
reduce  the  number  of  CNs  that  a  radiologist  must  inspect  (right).  The 
vascularity  expert  presumes  that  the  CNs  near  the  mediastinum  arise 
because  the  underlying  anatomic  structure  is  straight.  With  enough 
imagination  this  may  be  seen  in  the  image  processed  by  ANDS  without  the 
image  experts  (left),  where  all  CNs  are  displayed. 

The  experts  are  each  derived  from  appearance  models  of  the  objects  they  detect 
The  rib  appearance  model  is  based  on  my  interpretation  of  the  appearance  of  a 
section  of  a  rib  which  has  been  detected  as  a  candidate  nodule.  The  vascularity 
expert  is  based  on  the  concept  that  vascular  structures  (bronchi  etcetera )  and  their 
branches,  which  when  imaged  end-on  appear  as  CNs,  are  somewhat  striaght.  The 
nodule  expert  uses  features  which  were  derived  from  psychophysical  studies  of  the 
appearance  characteristics  which  distinguish  detected  from  undetected  nodules.  The 
success  or  failure  of  these  experts  depends  on  the  accuracy  of  the  models.  I  shall 
describe  each  expert  and  the  technical  approach  it  embodies. 

THE  IMAGE  EXPERTS 

Rib  Expert  -  Relational  Constraints  in  Parameter  Space 

The  salient  features  of  a  rib,  which  lead  to  its  detection  as  a  possible  nodule,  are 
given  in  the  appearance  model.  Relational  constraints  and  the  rib  expert  algorithm 
are  derived  from  die  appearance  model.  This  model  states  the  following  properties: 

-  a  rib  is  a  relatively  bright  object  bounded  by  two  parallel  edges; 

-  by  convention,  the  rib  edges  are  separated  by  180  degrees; 

-  the  width  of  the  rib  is  approximately  the  diameter  of  the  sought- 
after  nodule; 

-  the  rib  edges  are  approximately  centered  around  the  CN; 

-  the  parallel  rib  edges  are  the  strongest  (gradient  magnitude)  of  all 
edges  near  the  CN; 

-  these  edges  are  also  the  most  extensive  in  the  (windowed  -  64  x 
64  pixels)  image  of  a  CN. 


From  these  properties  the  relational  constraints  of  a  non-rib  and  a  rib  are 
derived.  The  parametric  equation  of  a  line  (Duda  and  Hart.  1972)  is  used  to  map 
candidate  rib  edge-elements  from  image  space  to  parameter  space,  P(p,  e): 

p  =  xcose  +  ysine  (Eq.  1) 

The  angular  orientation  of  the  edge  obtained  by  a  Sobel  edge  operator  is  used 
to  limit  the  range  of  0.  In  parameter  space  the  following  exclusive  relations  among 
location  and  values  of  peaks  must  exist  before  the  detector  responds  that  the 
parameterized  object  has  been  detected.  An  image  is  detected  as  a  non- rib  using  the 
following  relational  constraints: 

3  {(#»i»©i),  0>2’e2) .  (Pn*  en^  :  elJ  >=  /JMAX(P),  and 

P[p2>  02l  0MAX(P),  and 
PlPn-  0nl>=  ^MAX(P). 

where:  p  is  the  rejection  weight  (0  <  p  <  1). 

An  image  is  detected  as  a  rib  using  the  following  relational  constraints: 

3  {(pi.©i).  (p2’02^  :  0  ^  Ip l'rl  <  t,  and 

0  £  lp2‘f|  <  «•  and 
0  £  |0i-02  l8O|  <  ij,  and 
P(pj.0j]  >  aMAXfP),  and 
PIp:.021  >  aMAX(P). 

where:  e  and  11  are  error  tolerances  on  p  and  0,  respectively:  and  a  is  a  weight  (0  < 
a  <  1)  on  the  maximum  in  P. 

The  above  criteria  essentially  state  that:  a  non-rib  has  been  detected  if  more 
than  two  peaks  (in  parameter  space)  have  values  greater  than  the  rejection  threshold; 
and  a  rib  has  been  detected  if  only  two  peaks  whose  0  values  are  about  180  degrees 
apart  and  have  values  above  a  specified  threshold.  The  relational  constraints  are 
embodied  in  the  following  algorithm  which  operates  on  an  image  of  a  CN.  Here, 
successively  more  (and  weaker  in  magnitude)  edges  are  included  for  testing  using  the 
relational  constraints.  At  each  iteration  the  relational  constraints  are  tested;  first,  a 
test  attempts  to  reject  the  image  as  that  of  a  rib  and  another  test  attempts  to  accept  it 
as  an  image  of  a  rib.  The  output  of  the  rib  expert  is  used  by  the  nodule  expert. 


BEGIN  (•  Rib  Expert*) 

PctOfEdgesConsidered  =  2; 

WHILE(  PctOfEdgesConsidered  LE.  SomeMaximumValue ) 

BEGIN  (*  loop  *) 

(*  consider  all  edges  whose  gradient  magnitudes  are  in  the  top  PctOfEdgesConsidered 
of  all  gradient  magnitudes  *) 

IF(  TestForNonRib()  (*  if  most  prominant  edges  are  not  relatively  parallel  *) )  BEGIN 
return  NOT  RIB; 

END 

IF(  TestForRibO  (*  if  there  are  only  two  prominant  and  parallel  edges  *) )  BEGIN 
return  RIB-DETECTED; 

END 

PctOfEdgesConsidered  3  PctOfEdgesConsidered  *  2; 

END  (‘loop*) 
return  NOT-SURE; 

END  (*  Rib  Expert*) 

Performance  of  Rib  Expert 

The  rib  expert  was  tested  on  2750  CNs  whose  classifications  are  known.  Table  1 
illustrates  the  known  classifications  of  the  311  CNs  that  were  detected  by  the  rib 
expert. 

Performance  of  Rib  Expert  on  Entire  Database 
Taught  As: 

Rib  %  ri  SR  sv  Iv  SN  M.N  LN  lb  mb  SB  N1  un  Count 

Exp.  OK 

RIB  0.15  47  1  27  15  4  3  3  33  87  3  1  87  311 

213  ribs  were  taught 

311  CNs  were  classified  as  rib  by  the  Rib  Expert 

Table  1.  The  distribution  of  CNs  per  classification  (ri  =  rib:  SR  =  nodule 
on  rib;  sv,  Iv  =  small  and  large  vascularity,  respectively;  SN,  MN,  LN  = 
small,  medium,  and  large  nodule,  respectively;  SB  =  nodule  on  border; 

NI  =  nipple;  un  =  unclassified,  that  is,  did  not  fit  into  any  of  the  above 
classes)  of  the  311  CNs  that  were  detected  by  the  rib  expert. 

Note  the  rather  large  number  of  CNs  along  the  lung  borders  (lb  and  mb)  that 
were  detected  as  ribs.  This  suggests  a  modification  of  the  rib  appearance  model. 
Vertical  edges  (the  lung  borders)  should  be  included  in  the  relational  constraints  for 
a  non-rib.  Although  15%  of  the  CNs  that  were  detected  by  the  rib  expert  were  in  fact 
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classified  as  ribs  (true  positives),  22%  (47)  of  all  CNs  classified  as  ribs  (213)  were 
detected.  When  the  output  of  the  rib  expert  is  combined  w  ith  the  nodule  appearance 
features,  which  add  information  about  ribbiness,  computed  by  the  feature  extracter 
54%  (116)  of  all  ribs  are  correctly  identified  (sensitivity).  Overall,  62%  (116/187)  of 
all  CNs  classified  as  ribs  by  the  nodule  expert  were  in  fact  ribs. 

A  nodule  that  overlaps  a  rib  may  be  detected  as  a  rib  by  a  global  rib  detector 
[Ballard,  1978]  or  by  the  rib  expert  described  herein.  The  nodule  expert  augments  the 
information  about  ribbiness  provided  by  the  rib  expert  with  nodule  appearance 
features  which  aid  in  distinguishing  between  a  rib  and  a  nodule  on  a  rib.  The 
combined  use  of  the  rib  expert  output  and  the  nodule  appearance  features  by  the 
nodule  expert  resulted  in  the  correct  identification  of  50%  (7/14)  of  all  CNs  classified 
as  nodule  on  rib.  An  additional  36%  (5/14)  of  all  CNs  originally  classified  (taught)  as 
nodule  on  rib  were  classified  as  rib  by  the  nodule  expert.  Thus,  86%  (12/14)  of  all 
CNs  classified  as  nodule  on  rib  were  correctly  detected  as  some  sort  of  nodule. 

Vascularity  Expert  •  Back-projected  Hough  Transform 

The  appearance  model  for  the  vascularity  expert  is  rather  simple.  It  is  based  on 
the  notion  that  vascularity  originates  near  the  mediastinum  and  that  the  structures 
that  support  or  contain  the  vascularity  are  relatively  straight.  Thus,  the  expert  tests 
CNs  that  are  in  a  region  near  the  mediastinum  for  colinearity.  Each  CN  votes  for  a 
set  of  lines  which  may  pass  through  it.  This  voting  occurs  in  a  space  which  defines, 
by  Eq.  1,  a  set  of  parameterized  lines.  The  number  of  votes  at  a  peak  is  back- 
projected  onto  the  locations  of  the  CNs  in  image  space  that  gave  rise  to  the:  peak. 
Each  point  in  P  corresponds  to  a  line.  Points  with  the  most  votes  (peaks  in  P  space) 
correspond  to  lines  that  explain  the  most  evidence.  Thus,  the  vascularitv  expert 
computes  a  measure  of  colinearity  of  the  CNs  which  are  near  the  mediastinum.  This 
weight  is  an  input  of  the  nodule  expert. 

Performance  Evaluation 

Table  2  summarizes  the  performance  of  the  vascularity  expert  on  2750  CNs. 
The  database  contained  717  CNs  that  were  classified  as  vascularity:  105  (15%)  of 
these  were  detected  as  such. 


Performance  of  Vascularity  Expert  on  Entire  Database 


Taught  As: 


Vase  % 

ri 

SR  sv 

Iv 

SN 

MN  LN 

lb 

mb 

SB 

NI 

un 

Count 

Exp.  OK 

VASC  0.46 

16 

0  73 

32 

0 

0  1 

0 

71 

0 

0 

35 

228 

717  vascular  structures  were  taught 

228  CNs  were  classified  as  vascularity  by  the  Vascularity  Expert 

Table  2.  The  distribution  of  CNs  per  classification  of  the  228  CNs  that 

were  detected  by  the  vascularity  expert. 

Nodule  Expert  -  BMDP7M 

Radiologists  miss  25-30%  of  all  nodules  smaller  than  1.0  cm  [Garland,  1959; 
Yerushalmy,  1955].  Revesz  et.  al.  [1977]  studied  the  appearance  characteristics  that 
distinguish  undetected  from  detected  nodules.  These  features  in  addition  to  others 
(relative  distance  measures  from  the  top  of  the  lung  and  each  vertical  border,  and  the 
outputs  from  the  experts)  are  computed  and  used  by  the  nodule  expert. 

Twenty-three  features  are  computed  for  each  CN.  BMDP7M  [Dixon  and 
Brown,  1979],  a  commercial  statistical  package  for  computing  a  linear  discriminant 
function,  was  "trained"  with  1911  23-element  feature  vectors  whose  classifications 
were  specified.  This  package  was  used  to  determine  which  features  are  required  for 
discriminating  CNs  among  the  11  possible  classes  (of  Table  1).  BMDP7M  computed 
the  weights,  W  ,  and  constants,  c,  which  constitute  the  linear  discriminant  function,  d, 
for  a  feature  vector,  x,  where  dj(x)  =  xlWj  +  cj  [Duda  and  Hart,  1972].  Of  the  23 
computed  features  only  14  are  required  by  the  discriminant  function,  as  determined 
by  BMDP7M.  The  performance  of  the  nodule  expert,  which  characterizes  the 
performance  of  ANDS,  is  summarized  in  the  next  section. 

DISCUSSION 

The  three  experts  are  based  on  appearance  models:  a  local  appearance  model 
(rib);  a  global  structural  model  (vascularity);  and  the  physical  properties  of  a  nodule 
(nodule  features).  The  performance  of  an  expert  depends  on  the  accuracy  of  its 
model  and  the  proficiency  with  which  the  model  is  translated  into  an  algorithm. 
Thus,  each  descriptive  model  was  obtained  using  the  know  ledge  of  an  expert  human; 
the  rib  and  vascularity  experts  are  solely  based  on  my  knowledge,  while  the  nodule 
expert  is  based  on  appearance  characteristics  derived  by  observing  radiologists.  The 
image  experts  used  these  models  in  different  ways,  two  of  which  were  extensions  of 
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the  Hough  transform.  Because  of  the  difficulty  alone  in  qualitatively  describing  the 
difference  between  nodules  and  non-nodules,  no  nodule  appearance  model  and 
relatively  simple  method  (like  those  for  ribs  and  vascularity)  of  detection  could  be 
devised.  The  appearance  characteristics  used  by  expert  human  nodule  detectors  were 
used  instead  in  a  traditional  pattern  recogniton  system. 

Performance  of  the  rib  expert  may  be  improved  by  adding  the  following  to  the 
appearance  model  and  updating  the  relational  constraints  for  a  non-rib:  non-ribs 
contain  nearly  vertical  parallel  edges.  Thus,  a  CN  with  nearly  vertical  parallel  edges 
would  be  rejected  as  a  rib. 

The  vascularity  expert  fails  because  it  is  based  on  an  unsound  model.  Vascular 
structure  is  not  straight  and  the  CNs  that  arise  because  of  it  do  not  lie  in  a  line. 
Close  inspection  of  Fig.  2  (pre-expert  image)  may  reveal  this.  Such  a  notion  of  co- 
linearity,  however,  may  be  useful  in  reasoning  abo.  CNs  that  lie  on  a  rib,  which  is 
essentially  a  linear  structure. 

The  success  of  the  experts  can  be  evaluated  in  two  ways:  by  comparing  the  :11m 
false  negative  rate  of  ANDS  with  and  without  the  experts;  and  by  comparing  the 
number  of  CNs  that  a  radiologist  must  sift  through  before  being  confident  of  having 
inspected  a  CN  that  was  reported  by  ANDS.  ANDS  was  tested  with  and  without  the 
experts  on  37  chest  radiographs  that  contain  at  least  one  nodular  abnormality.  Two 
films  with  nodules  were  mis-diagnosed  by  the  unexpert  system  and  four  were  mis¬ 
diagnosed  by  the  system  that  used  the  experts;  the  corresponding  true  positive  rates 
are  92%  and  86%.  Although  these  rates  are  higher  than  those  of  radiologists,  no 
statistically  valid  comparasion  may  yet  be  drawn.  The  total  number  of  CNs  that  a 
radiologist  must  inspect  (in  order  to  be  99%  confident  of  having  inspected  a  detected 
nodule)  drops  from  12  to  4  as  a  result  of  using  the  image  experts.  Thus,  the  image 
experts  reduce  the  number  of  points  that  a  radiologist  must  inspect. 

Performance  of  the  nodule  expert  may  presumably  be  improved  by  training 
with  additional  radiographs  which  contain  nodules. 
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